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Iterated games are a fundamental component of economic and evolution- 
ary game theory. They describe situations where two players interact repeat- 
edly and have the possibility to use conditional strategies that depend on the 
outcome of previous interactions. In the context of evolution of cooperation, 
repeated games represent the mechanism of reciprocation. Recently a new 
class of strategies has been proposed, so called 'zero determinant strategies'. 
These strategies enforce a fixed linear relationship between one's own payoff 
and that of the other player. A subset of those strategies are 'extortioners' 
which ensure that any increase in the own payoff exceeds that of the other 
player by a fixed percentage. Here we analyze the evolutionary performance 
of this new class of strategies. We show that in reasonably large populations 
they can act as catalysts for the evolution of cooperation, similar to tit- for- tat, 
but they are not the stable outcome of natural selection. In very small popu- 
lations, however, relative payoff differences between two players in a contest 
matter, and extortioners hold their ground. Extortion strategies do partic- 
ularly well in co-evolutionary arms races between two distinct populations: 
significantly, they benefit the population which evolves at the slower rate - 
an instance of the so-called Red King effect. This may affect the evolution of 
interactions between host species and their endosymbionts. 



Introduction 

The Iterated Prisoner's Dilemma (IPD) has a long history of serving as a model 
for the cultural and biological evolution of cooperation (li|-|9|). A new class of so 
called zero-determinant (ZD) strategies has recently attracted considerable atten- 
tion (flo[ [ill 29). Such strategies allow players to unilaterally enforce a linear 



relation between the own and the co-player's payoff. A subset consists of the so- 
called equalizer strategies: these assign to the co-player's score a predetermined 



value, independent of the co-player's strategy, see also (|12l) . Another subset con 



sists of the extortion strategies: they guarantee that the own surplus exceeds the 
co-player's surplus by a fixed percentage. Press and Dyson (|ld) have explored 
the power of ZD-strategies to manipulate any 'evolutionary' opponent, i.e., any 
co-player able to learn, and to adapt. 



In their commentary to Press and Dyson, Stewart and Plotkin (|11|) ask: 'What 
does the existence of ZD strategies mean for evolutionary game theory: can such 
strategies naturally arise by mutation, invade, and remain dominant in evolving 
populations?' From the outset, it may seem that the opportunities for extortion 
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strategies are limited. If a strategy is successful, it will spread, and therefore be 
more likely to be matched against its like: but any two extortioners hold each other 
down to surplus zero. However, if the two players engaged in an IPD belong to 
distinct populations, the evolutionary prospects of extortion improve significantly. 

In the following, we investigate the impact of ZD-strategies on evolutionary 
game theory. We show that in well-mixed populations, ZD-strategies can play 
an important role, but only as catalyzers, not as long-term outcome. However, 
if the IPD is played between members of two separate populations evolving 
on different time-scales, extortion strategies can get the upper hand in whichever 
population evolves more slowly, and enable it to enslave the other population, an 
interesting example of the so-called Red-King effect (13;). 

The Prisoner's Dilemma (PD) game is a game between two players / and // 
having two strategies each, which we denote by C ('to cooperate') and D ('to 
defect'). It is assumed that the payoff for two cooperating players, R, is larger 
than the payoff for two defecting players, P. If one player cooperates and the 
other defects, the defector's payoff T is larger than R, and the cooperator's payoff 
S smaller than P. Thus the game is defined hy T > R > P > S. An important 
special case is the so-called donation game, where each player can 'cooperate' 
(play C) by providing a benefit b to the other player at own cost c, with < c < 6. 
Then T = b, R = b - c, P = and S = -c. 

In the Iterated Prisoner's Dilemma game (IPD), the two players are required 
to play an infinite number of rounds, and their payoffs Pj resp. Pjj are given by 
the limit in the mean of the payoffs per round. An important class of strategies 
consists of so-called memory-one strategies. They are given by the conditional 
probabilities pr, ps, px and pp to play C after experiencing outcome R, S, T resp. 
P in the previous round. (In addition, such a strategy has to specify the move in 
the first round, but this plays no role in the long run, see e.g. h4)). An important 
class of memory-one strategies consists of reactive strategies, which only depend 
on the co-player's move in the previous round (not the own). Then pr = pr =: p 
and pp = Ps =: q, so that a reactive strategy corresponds to a point (p, q) in the 
unit square (|l5h . 

We will first define and characterize zero-determinant strategies, equalizers 
and extortioners. We then investigate, in the context of evolutionary game the- 
ory, the contest between extortioners and four of the most important memory-one 
strategies. We will show that extortion cannot be an outcome of evolution, but can 
catalyze the emergence of cooperation. The same result will then be obtained if 
we consider all memory-one strategies: in particular, ZD-strategies can only get 
a foothold if the population is very small. If the IPD is played between members 
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of two distinct populations, ZD-strategies can emerge in the population which 
evolves more slowly. In particular, extortion strategies can allow host species to 
enslave their endosymbionts. 

Methods and Results 

Definitions. Press and Dyson (fiol) define the class of 'zero-determinant' strategies 
ZD as those memory-one strategies {pr,Pt,Ps,Pp) satisfying, for some reals 
a, 13, 7, the equations 

Pn-l = aR + /3R + 'y [la] 
PS -I = aS + /3T + -f [lb] 
Pt = aT + (3S + -f [Ic] 
Pp = aP + (3P + -f. [ld\ 

(We note that l—pp. and 1—ps are the probabilities to switch from C to D, while 
Pt and pp are the probabilities to switch from D to C.) Press and Dyson showed 
that if player / uses such a ZD strategy, then 

aPi + (3Pn + 7 = 0, [2] 

no matter which strategy player // is using. Equalizer strategies are those ZD 
strategies for which a = ^ /3: then 

Pii = -l/P. [3] 

Thus player I can assign to the co-player any payoff between P and R. (Indeed, 
since the Pi have to be between and 1, it follows that (3 < and P < P^ < R). 
The so-called x-extortion strategies are those ZD-strategies for which 7 = — (a + 
P)P, with X = > 1- Then 

P,-P = x{Pn-P). 

In this case, player / can guarantee that the own 'surplus' (over the maximin value 
P) is the X"fold of the co-player's surplus. 

Press and Dyson speak of zero-determinant strategies because they use for 
their proof of [2] an ingenious method based on determinants. In Appendix A, 
we present a more elementary proof, following (fl2l) . Within the four-dimensional 
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unit cube of all memory-one strategies (pb^Ps^Pt^Pp), the ZD strategies form 
a three-dimensional subset ZD containing the two-dimensional subsets £Q and 
£X of equalizers resp. extortioners (see Appendix B). In Fig.l we sketch these 
sets for the reactive strategies. 

Extortion within one population. In order to investigate the role of extor- 
tion in the context of evolutionary games, we concentrate on the donation game, 
but stress that the main results hold more generally. We first consider how a x- 
extortion strategy fares against some of the most important memory-one strate- 
gies, namely TFT = (1, 0, 1, 0), AllD = (0, 0, 0, 0), AllC =(1,1, 1, 1) and the 
Win-Stay-Lose-Shift strategy WSLS which is encoded by (1, 0, 0, 1), and hence 
cooperates if and only if the co-player's move, in the previous round, was the same 
than the own move, see Q). For the donation game, the payoff for a player using 
strategy i against a player with strategy j is given by the (i, j)-th element of the 
following matrix: 
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All D 
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[4] 



Let us start with the pairwise comparisons. E^, is neutral with respect to AllD. 
It is weakly dominated by TFT. (A TFT-player does not better than an extor- 
tioner against extortioners, but interactions with other TFT -players are giving an 
advantage to TFT.) AllC players can invade extortioners, and vice versa: these 
two strategies can stably coexist in proportions c(x — 1) : (6 + c). Finally, WSLS 
dominates extortioners. We note that the mixed equilibrium of extortioners and 
unconditional cooperators can be invaded by each of the other three strategies. 
The same holds for the mixed equilibria of extortioners and unconditional defec- 
tors, if the frequency of extortioners is sufficiently high. In particular, TFT can 
always invade such a mixed equilibrium, but can, in turn, be invaded by WSLS or 
AllC. No Nash equilibrium involves E^. If 6 < 2c, there are two Nash equilibria: 
a mixture of TFT, AllC and AllD and a mixture of WSLS, AllC and AllD. If 
b > 2c, there exist four Nash equilibria. In particular, WSLS is then a strict Nash 
equilibrium. 



The replicator dynamics (see, e.g., (|14l . ll6l ll7[)) displays for the payoff matrix 



5 



continuous families of fixed points and periodic orbits, and hence is far from be- 
ing robust. The same applies to most other deterministic game dynamics. It seems 
more reliable to consider a stochastic process which describes a finite, well-mixed 
population consisting of M players, and evolving via copying of successful strate- 
gies and exploration, i.e., by a selection-mutation process (see, e.g., (inLllSLIllI) ). 
Selection is modeled as an imitation process; in each time step, a randomly cho- 
sen individual A adopts the strategy of a role model B with a probability which 
increases with the model's payoff. Additionally, mutations occur with a small 
probability /i > (corresponding to the random adoption of another strategy), 
thereby ensuring that all population states can be reached by the evolutionary dy- 
namics. 

Any such stochastic process yields a steady state distribution of strategies. We 
find that while extortioners are never the most abundant strategy, they can play the 
role of a catalyzer. Indeed, if only AllD and WSLS are available, a population 
may be trapped in a non-cooperative state for a considerable time, leading to a 
mutation-selection equilibrium that clearly favors defectors (see Fig. 2A). In such 
a case, extortioners (Fig. 2B) and TFT (Fig. 2C) offer an escape: both strategies 
can subvert an AllD population through neutral drift. Once defectors are rare, 
WSLS outperforms TFT, and it also prevails against extortioners if the popu- 
lation is sufficiently large (in a direct competition, WSLS always gets a higher 
payoff than E^if M > 1 + x)- Thus, in large populations, extortioners and TFT 
tip the mutation- selection balance towards WSLS. In contrast, expanding the 
strategy space by adding AllC has a negligible impact on the equilibrium (Figs. 
2D and 2E), see also (19^. 

What happens when players are not restricted to the five specific strategies, but 
can choose among all possible memory-one strategies? We study this by using the 
stochastic evolutionary dynamics of i2&), assuming that mutants can pick up any 
memory-one strategy, and that the mutant reaches fixation, or is eliminated, before 
the next mutation occurs. Overall, this stochastic process leads to a sequence 
of monomorphic populations. The evolutionary importance of a given strategy 
can then be assessed by computing how often the state of the population is in its 
neighborhood. For a subset A of the set of memory-one strategies, we denote the 
(5-neighborhood of A (with respect to Euclidean distance) by A^, and let iJi{As) 
denote the fraction of time that the evolving population visits As. We say that 
As is favored by evolution if the evolutionary process visits As more often than 
expected under neutral evolution, i.e., if iJi{As) is larger than the volume of As. 
We apply this concept io A = ZV.SQ, EX. 

Extensive simulations indicate that neither extortioners, nor equalizers or 
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zero-determinant strategies, are favored by evolution if the population is reason- 
ably large (see Fig. 3A). By contrast, very small population sizes promote the 
evolution of these behaviors. For extortioners, this result is intuitive: in small 
populations, relative payoff advantages determine the fate of a strategy, rather 
than absolute payoffs (|18b : this effect may even result in the evolution of spite, see 
(l22ll23|) . It is less clear why the other two behaviors are abundant in small popula- 
tions, and why large population sizes lead to their decrease. However, simulations 
show that the decline of zero-determinant strategies and equalizers is partly due 
to the success of WSLS-like strategies, which soon dominate the evolutionary 
dynamics (see Fig. 3B): as the population size increases, individuals prefer strate- 
gies that cooperate after mutual cooperation and after mutual defection, and that 
defect otherwise. As a (possibly surprising) consequence, larger populations also 
yield higher average payoffs (Fig. 3C). These qualitative results are robust with 
respect to changes in parameter values, such as benefits and costs, or the strength 
of selection, indicating that WSLS, rather than extortion, is favored by evolution 
as soon as the population size exceeds a critical level. 

Extortion between two populations. Let us now consider two species (for 
instance, hosts and their symbionts), or two classes of a single species, old and 
young, for example, buyers and sellers, or rulers and subjects, engaged in an 
IPD game which, of course, is now unlikely to be symmetric. In such situ- 
ations, extortioners may evolve even in large populations. Indeed, extortioners 
provide incentives to cooperate: as shown by Press & Dyson (llOh . AllC is always 
a best response to an extortion strategy. In a single population of homogeneous 
players, this is not turned to advantage, as the extortioners' success leads to more 
interactions with their own kind. If extortioners evolve in one of two separate 
populations, they will not have to interact with co-players of their own kind. Nev- 
ertheless, their success may be short-lived, since they will be tempted to adopt the 
even more profitable AllD-strategy as a reaction to the AllC co-players which 
they have produced. 

To achieve lasting success in a two-population set-up, extortioners need to be 
stubborn, and cling to their strategy. To elucidate this point, we extend our pre- 
vious analysis by revisiting a co-evolutionary model of Damore and Gore (l26h . 
These authors consider host-symbiont interactions where each host interacts with 
its own subpopulation of endo-symbionts. Let us assume that these interactions 
are given by an IPD game. Members of both species reproduce with a probabil- 
ity proportional to their payoffs, replacing a randomly chosen organism of their 
species. However, the two populations of hosts and symbionts may evolve on dif- 
ferent time scales, as measured by their relative evolutionary rate (RER). For a 
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relative evolutionary rate of one, hosts and symbionts evolve at a similar pace in 
the evolutionary arms race, and no population is able to extort the other (Fig. 4A). 
This changes drastically as soon as we increase the relative evolutionary rate, by 
allowing symbionts to adapt faster. Fast adaptation results in a short-term increase 
of the symbionts' payoffs, since they can quickly adjust to their respective host. In 
the long term, however, hosts learn to adopt extortion strategies (Fig. 4B), thereby 
forcing their symbionts to cooperate. Thus it pays in the long run, for the host, 
to be pigheaded and refuse to adapt; for the parameters in Fig. 4B, the resulting 
equilibrium allocates them on average more than 90% of the surplus. 



Discussion 

Our main results show that within one population, extortioner strategies can act 
as catalyzers for cooperation, but prevail only if the population size is small; and 
that in interactions between two populations, extortion can emerge if the rates 
of evolution differ. This holds not only for the donation game (and therefore 
whenever R + P = T + S), but in considerably more general contexts. We could 
assume, for instance, that the players alternate their moves in the donation game 



(1241 1251); or that the underlying PD game is asymmetric (the definitions have to 
be modified in an straightforward way). As noted in [id), some results hold also 
for non- PD games; this deserves further investigation. 

Extortion strategies are only a small subset of ZD-strategies. We have shown 
that within large populations, the class of ZD strategies is not favored by selec- 
tion, in the sense that its neighborhood is not visited dis-proportionally often. This 
does not preclude, of course, that certain elements of this class are favored by se- 
lection. Thus Generous TFT (1,1 — c/6, 1,1 — c/6) does well. So do other, less 
known strategies. In particular, Stewart and Plotkin highlighted a class of strate- 
gies defined, instead of Eq. [3], hy Pj — R = x{Pn — R) (with x > !))• A player 
using this strategy does not claim a larger portion of the 'surplus', but a larger 
share of the 'loss' (relative to the outcome R of full cooperation). Remarkably, 
these 'compliant' strategies do as well as WSLS. 



In a preprint by Adami and Hintze (|29t) . the evolutionary stability of several 



ZD strategies was tested by replicator dynamics and agent-based simulations, 
which independently confirms the result that these strategies do not prevail in 
large populations. They used payoff values _R = 3, 5 = 0, T = 5 and P = 1, i.e., 
a Prisoner's Dilemma game which cannot be reduced to a donation game. Adami 
and Hintze also discuss the evolutionary success of 'tag-based' strategies, which 
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use extortion only against those opponents who do not share their tag. These 
strategies are not memory-one strategies, but use them in specific contexts. 

In (|10|) . Press and Dyson analyzed ZD strategies in the context of classical 
game theory, with two players locked in contest: extortion strategies play an im- 
portant role, as do the more orthodox trigger strategies, see (|3i|6|). In the context 
of evolutionary game theory, whole populations are engaged in the game. Not 
surprisingly, for very small population size extortion strategies still offer good 
prospects. In larger populations (with our parameter values, for M > 10), this 
is no longer the case. However, evolutionary game theory can reflect features of 
classical game theory if the two interacting players game belong to two separate 
evolving populations. 

Therefore, extortion may evolve in endosymbiotic relationships due to the so- 
called Red-King effect (|13L l27L l28b: the species that evolves at a slower rate gains 
a disproportionate share of the benefits. This requires two conditions to be met: 
individuals need to come from different populations, and these populations have 
to evolve on different time scales. If these conditions are fulfilled, extortioner 
hosts can manipulate their symbionts' evolutionary landscape in such a way that 
the host's and the symbionts' payoffs are perfectly correlated. This ensures that 
only those symbiont mutants can succeed that are beneficial for the host. In this 
sense, such hosts apply an evolutionary kind of mechanism design; they create an 
environment that makes the symbionts' cooperation profitable for the symbionts, 
but even more profitable for themselves. 



Appendix A: Proof of Eq. [2] Let us denote by Pi{n) and Pn{n) the player's 
payoffs in round n, by Si{n) the probability that / experiences outcome i G 
{R, S,T, P} in that round and by qi{n) the conditional probability, given that 
outcome, that // plays C in round n + 1. By conditioning on round n, we see that 
sji{n + 1) is given by 

SR{n)qR{n)pR + ss{n)qs{n)ps + ST{n)qT{n)pT + sp{n)qp{n)pp, 
and ss{n + 1) by 

SR{n){l-qR{n))pR+Ss{n){l-qs{n))ps+ST{n){l-qT{n))pT+Sp{n){l-qp{n))pp. 

Using (1), and setting := {R,S,T,P), g// := {R,T,S,P), we see that 
w{n) := SR{n + 1) + ss{n + 1) - {sR{n) + ss{n)) is given by 

as{n) ■ gj + (3s{n) ■ gjj + 7s(ra) ■ 1 
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which is just aPiiji) + f3Pjf{n) +7. Summing overn = 0, 1, — 1 and 
dividing by N, we see that 

MN) + ss(N) - - ssiO) ^ 

and hence Eq. [2] holds, independently of the strategy of player //. The same 
proof works for any 2x2 game (even if it is asymmetric: one just has to replace 
g// with the corresponding payoff vector). In many cases, however, there will be 
no solutions to Eq. [1] which are feasible (i.e., probabilities between and 1). 

Appendix B: the sets ZT>, EQ. and EX\ Elementary algebra shows that within 
the four-dimensional unit cube of all memory-one strategies {pr,Ps,Pt,Pp), the 
ZD-strategies are characterized by 

{1-Pr){S + T-2R) + {1-ps){R-P)+Pt{P-R)+Pp{S + T- 2P) = 0, 

(a three dimensional subset of the cube). Equalizers are characterized, in addition, 
by 

{R - P){ps-PT - 1) = (T - S){pR-pp - 1), 
(they form a two-dimensional set) and x-extortion strategies hy pp = and 

Pt[S + x{T - P)] = (1 - Ps)[T + x{P - S)], 

(for each x a one-dimensional set). In the special case of the donation game, these 
equations reduce to 

Pr + Pp = Ps+Pt, 

{b - c){ps - Pt - 1) = (& + c){pr -Pp - 1), 

Pt{c + xb) = (1 - Ps)ip + Xc), 

respectively. The set SQ oi equalizers is spanned by (1, 1, 0, 0), (c/6, 0, c/6, 0), 
0, 1, 1^) and (1, 1 — c/6, 1,1— c/h), the set £X of extortion strategies by 
(1, 1, 0, 0), (c/fe, 0, c/6, 0) and (1, 0, 1, 0). All reactive strategies are ZD-strategies, 
the reactive equalizers are those satisfying p — q = c/b, and the reactive X' 
extortioners those with g = and p = {b + xc) /(c + xb) (see Fig.l). 

Acknowledgement: KS acknowledges support by Grant #RFP- 12-21 from Foun- 
dational Questions in Evolutionary Biology Fund. 
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Figure 1: Reactive strategies (pb=Pt=P,Ps = Pp = Q) for the donation game. 
All reactive strategies (the square < p, g < 1) are ZD strategies. The equalizer 
strategies are those on the segment between 'generous TFT' (p = 1, q = 1 — c/b, 
see (15)) and 'Miser' (p = c/b, q = 0, see dzll)), the extortion strategies those 
between 'Miser' and TFT (p = 1, q = 0), and the 'compliant' strategies (see 
(|11|) and Discussion) those between 'generous TFT' and TFT. 
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Figure 2: Evolutionary competition between some important strategies in the 
IPD. For various population sizes M, the graphs show the frequency of each 
strategy in the mutation- selection equilibrium. We consider two mutation regimes, 
the limit of rare mutations — )• (top row), for which the equilibrium can be cal- 
culated analytically, and a regime with mutation rate fi = 0.05 (bottom row) which 
is explored by individual-based simulations. For the copying process, we assume 
that individuals A and B are chosen randomly. A switches to B's strategy with a 
probability given by (1 + ^cp[s{Pa — Pb)])^^, where s > corresponds to 'se- 
lection strength', see e.g. (|2lh . If AllD competes with WSLS the population is 
mostly in the defector's state, independent of population size and the mutation rate 
(A). However, once -E^ or TFT is added, WSLS succeeds for sufficiently large 
populations (B and C). Adding AllC only leads to minor changes in the stationary 
distribution (D and E). Parameters: 6 = 3, c = 1, s = 1, and x = 2. 
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Figure 3: Statistics of the evolutionary dynamics for memory-one strategies for a 
range of different population sizes. We have calculated (A) the relative abun- 
dance of extortioners, equalizers, and ZD strategies, i.e. the time spent in a 
5-neighborhood, divided by the volume of that neighborhood; (B) the average 
strategy of the population; (C) the average payoff. Extortioners, equalizers and 
ZZ^-strategies are only favored for small population sizes. As the population size 
increases, individuals tend to apply WSLS-\\kQ strategies, and to cooperate only 
after mutual cooperation or mutual defection. As a result, the average payoff 
increases with population size. For the simulations, 10'' mutant strategies were 
randomly drawn from the space of memory-one strategies. The switch from a 
monomorphic population using strategy X to a monomorphic population using 
strategy Y occured with the probability of fixation of a single Y mutant in a pop- 
ulation of X-residents. Parameters: 6 = 3, c = 1, 5 = 0.1 and s = 100. 
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Figure 4: Evolution of extortion in host-symbiont interactions. The graphs show 
two typical simulation runs for a population of 80 hosts, each having a subpopula- 
tion of 25 symbionts. For each simulation run, the upper graph shows the average 
payoff for each population, whereas the lower graph shows the Euclidean distance 
of each population to the set of extortioners (which can be at most 1.5275). In the 
initial population all individuals cooperate unconditionally. The further evolution 
depends on the relative evolutionary rate: (A) If RER = 1, both species converge 
towards AllD, and no population is able to extort the other. (B) For RER = 500, 
symbionts evolve much faster. In the short term, they can thus increase their av- 
erage payoff by switching to a non-cooperative strategy. However, in the long 
term hosts apply extortion strategies to force their symbionts to cooperate. Even- 
tually, the hosts' payoff exceeds b — c, whereas the symbionts' payoff is close to 
zero. To model the evolutionary process, we followed (|26b : Whenever a symbiont 
reproduces, its offspring remains associated with the same host. Whenever the 
host reproduces, the new host offspring acquires its symbionts from other hosts 
(horizontal transmission). Mutations occur with probability /i = 0.05, by adding 
Gaussian noise to the memory-one strategy of the parent (a = 0.05). The process 
is run for 1, 000 host generations (corresponding to more than 10^ reproduction 
events for RER = 1, and more than 10^ reproduction events for RER = 500). 
The other parameters were b = 3, c = 1 and s = 10. 
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